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ABSTRACT 

A pilot study was initiated to achieve reliai)ility 
and especially validity indexes for leasuting *;cesearch and 
development impact. Tvo opposing questionnaire were administered to 
each of thr^e target groups: 164 high school vocational education 
students, 120 secondary school vocational education teachers, and 5 6 
vocational education administrators. HaliE of each group received the 
original questionnaire measuring product impact; half received the 
alternative questionnaire measuring product nonimpact,. Three impact 
questionnaires, developed primarily by vocational education 
administrators, were constructed through item analysis. Nonimpact 
questionnaire's stated the impact questions negatively. The student 
questionnaire contained 16 items, the teacher questionnaire 36 items, 
and the adminisrtrator questionnaire 20 items. The following are 
original and alternative questionnaire discriminant validity 
correlations and rqliability correlations for the three groups: 
students: -.049 validity, .83 reliability; teachers: -.18 validity, 
•78 reliability; and administrators: -.45 validity, .86 reliability. 
Better impact assessment was found in the administratdf^ domaxn tiian 
in the^ student and teacher domains. Each adjusted questionnaire 
contains teii acceptably valid items. An example of sensitive/valid 
items from each domain is (1) students — gaining nev knowledge from 
^the product, (2) teachers**the completeness of details covered in tlie 
product, and (3) administrators--the absence of many better 
alternative products in the subject area. (CSS) 
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VOCATIONAL EDUCATION .R&D PRODyCTS 
IMPACT, : RELIABILITY , AND VALipiTY 



I 



;Lynn.' Ji;%arris"'' 



Although the" CQinmittee on .Vocatioiiai Education. Research 
and Development (1976). has pointed out a lack o^ documented 

• ^ • •■ ' Z ' . " ■ :^ ' ^ 

R&D impact, it is essi^t^ial that if such docvotientation is 
obtained, th6_afs,dssment tool should dembnstrate some degree ^ 
of reliability an3 validity, the latter being morfe important. 
Even the studies cited by COVERD as ,being Acceptable examples 

of documented impact show no evidence of reliability or validity ' 

■ ^ , - ■ ■ \ ' i ; . / 

within their methods of assessment. Instead, such studies (All^h, " - 

1968!;^ elm and Boerrigter, 1974; Roney, 1971; Sutherland AssogiAtes, 

-•' ' . » * /■ ' ' - ■ ^ ■ * / • *^ ' . 

1974;/ari<itSouthwide Research Cjo^rdinating Coilpcil, 1975) gav6 . 

rather abstrac£^ nonempirical accoxants of percseived impact; further- 

more, all studies seemed to be directed, towaxm R&D projects arid 



processes that are set apart from 'the ifssue of ^ R&D products « One 
exception, however, is Crawford (19^2), Who specaJfi^nai*^^ studied 
the impact of R&D products. But his research, was hbt limited to 
vocationjil education; nor did it have any consistent reliability 
or validity of impact measurement on an overall basfis^* fThe Crawford 
sl:udy resembled the research cited in the COVERD" repot t by listing 
/characteristics of products perceived to be ex^ih^l^rs ^of positive 
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TMI$ OOCUMBNT MAS BEEN REPftO- f 
OOCIO EXACTLY AS RECElVEO FROM \ 
THE PERSON OR ORGANIZATION ORIGIN. . / * 

ATING IT POINTS OP VIEW OR OPINIONS * ' TO THE EDUCATION/ - 

«:;i%'^?,:rN:T^§!!rtr.'s^^T^^^^^^^^^^ p ' • ' information center (er,ci and. 

tooc*TiON fosiT.oN9R»^!)LVcr .; users of the eric system," 



TO THE EDUCATION AC^ESOURCES ' 



impactf^ apparent advantage of th^ Craviord istudy is that * / 

selected products wexe systeinatiQally ^gxd.uped'^ to describe 



r 



different tyt)es of existing positive; impafct; * ^ 

It "seems reasonable to expect- that the introduction of a ^ 
product into a vo||ational education 6^ting would result in a' 
fayorabie state of^ ^f fair's ^ but Dick (197e) points out that ^ 

negat^^ve i^pa^zt' 6ou:j^ t>ccut also. For the sake of the prfersent 

•* ■ •>' •'' ■ ' ' ' ^ »' 

study/ impact is op,ejdM:ioni^lly def indd as a mea^surable change 

r / ' t ' ' ? ' ■ ' ' • 

resulting from the introduction of' 4 pfpduct into .a voca'^^ional ' 

education setting. This differs sharfJly from •'other definitions 

of impact, such as Miller and Miller's (1974) "maintenance and 

extension" viewpoint. . ^ . 

The main objective of thi^ pilot study is ^to achieve sontfe 

f ■ £ ^ ■ ^ ''' ^ ■ ' 
index gf measuring impact wit^ evidence of ireliability and 

■ ■ ' ^ ^ ' • . ■ ' ., / — ^ " * . > ■ 

validity, the lat^ter being the most ijthfjprta^t -a^:. this early ° 

/■ \ •• . .. .r,' ■ jy^'-i / -^^ ^ . 

stage of the a;e9earch effort.v The val^jdirtSr of *ih^asyreinent/ in ^ 
the case of a fairly new concept suclj;a^ impact //^can be very 
troublesome, but reliability is npt . ei<tteiheiy diff ifeul't t6 attaii 
In the present study/ ^he author has -chQaifeh^to app^roa'qh this 
problem by establishing coefficients of disqyimiriant validity 
demonstrated by Selltia, Wrightsman,^ and Cook (1976) . That is 
to say, since there i^'so littlje known about the impact of . these 

■ • ^ ' y f ' * - ■ ■ ^ ' ' . ■ ^ - 

R&D products> the reseaifch must b^e Started from a very Da^ii.c 



standpoint. f J 

^ Purpose 

^^^^^^ 

This pilot study produced valid instrumentation which can 
be used to 'assess the impact of R&D products in general, with 



resect to^studeptS/ teacheirs, and .administi;ators; , The ra^ioiaa^^ 
for achieving this task was characterized by . the notioji of (^iscTti^ . 
minant validity (Selltiz, V^ightsman, and Cook, 19V76) . In other 
wor^s, it was necessary to measure impact by two opposing ^ques- 
tionnaires administered ^o each of three target gifoups (students, 
teachers, and administrators) . One (original) qyistionnaire was 
constructed to measure impact of products. Ano/her (alternative) 
quesionnaire was consltructed to measure nonimpact* of prbdilcts. 
"^t was assumed that if ther^e was little or no correlation, or a 

••' ' , ■■ ^ . . ■■ - --v - 

negative correlation , between the- original and alter»lative ques- 
f tionnaires, this would indicat,© -that the origina^l questionnaire 

was indeed getting at attitudes Sspecifiaally directed toward imp<act. 



Method A ' ^ 



' subjects . . 

" ' ' .. \ . 

. 4 Students: One hundred sixty -sfour voca?tional education stiodents 

"'5^. ' . \ . ^ ^ - . ^ 

^ eiu:olle4 in ^hree^ publ4.c schools were asked to express' their 

opijiions of the, particular R&D product they were using (a standard 

set" of core curriculum materials fc^r each student's course of 

' study) ;^ ^ All respondents were midple /income high school, students 

from five geographical locations, inciluding/Water Valley and* Oxford, 

Mississippi; and Stillwater, Tulsa, jand Oklahoma City, Oklahoma. , 

' The students v^re earolled- in severa'l ^dlf^erent courses /of st^dy, 

e.g.r home economi^QS, food service, ^to mechanics, vocational 

' ' ■ # / ^ ' I ^ ■ ' ' . 

agriculture, smiil^ engine rej^air, el€;ctrpni/cs, et;c. EJach student 

/possessed his/her own set of cpre stiidsC, Materials whiqh had^B^febn.^ ' 

. / - ■ . • * ^ ^ . * ■ r ' \ 

issued^, at thet bj^ginning of 'She current schooi year. 

>..■ •/ . > ^ / • . 
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Teachers: One hundred twenty vocational education teachers 
were aXso asKed to express their ojjinions^of the partitular R&D 
product th^y were us;^fWffpta standar'd set of core curriculum 
materials fo^^ach teacher's respec^tive course of instruction). • 
pach^ teacher was employed in a public school system located in 

» » • '' .'*' 

the same geograpljical areas as the students listed above^ and 
each teacher, ta^lght at least one of the courses in which the ' 
vocational Jdue^ students were enrolled (listed above) 

Administrators: Fifty-six administrators from the statpe 
(^^klahomk/ Mississippi ^ and Georgia were also asked to exp^^ess 

their opinioas of the specific R&D product tl^ey were using./ The 

' ■ ^ ■' ■ 

product consisted of vocational education core curriculum , ' 
materials and a set of pifocedufes for evaluating 'Vocational 
education programs within ^secondary school systems. Admiiiistrjat 
were state, employees^ ^ith the exception of a few who provided/ 

admirtistrative services at the locail level. ^ \ 

/ / ^ " ■ ' ^ - . > ■ '/ 

Ipstrumentation ' * 

Thxee impact questionnaires were constructed as a' resiilt of 
an vitem analys-is conducted at the National Center for Resecurch 
in Vocational Education, located at The Ohia State Uniyersity. 
panel of six experts^from various geographical locations was ' 

assembled to rate each item In* three, separate item pools: an item 
pool £or - students / an item pool for tejaclliers/ and an item pool 
for administrators. Items were generated by Nati^iial Center 
researchers, vocational education teachers, and Vocational 
education students on the basis of theiV experience. Part of the 



item pool was also obtained from a study by McCasIin (1973) . 

An arbitrary cuftff point was established for each item 
pool in order to retain or discard items. Consequently^ 
three questionnaires were constructed based upon items which 
had a mean racing above the established cutoff points. Inter--^ 
rater reliabiity coefficients were then computed for each ques- 
tionnaire. The inter-rater reliability qotf f icients for each 



n 
} 



questionnaire yielded .21 for students, 02 for teachers and 

.61 for' administrators. The reason lor such a low amount of 

agreement Qn the student* and teacher ^^iikstionnaires, perhaps, 

w^s that all six of the selected experts were vocatit>nal jBiSucai^ion 

administrators, who agreed more^^when tlley^rated items dealing witl) ^ 

administrartive impact than with either student jbr teacher impact. 

After the undesirable items were discarded) tlje student 

questionnaire contained 16 items, * the teacher questibnnaire con-'^ 

tained 36 items, and *the administrator questionnaire contained 

' - . 4 

20 items. All-^questionnaites wer^ .formed into Like^' scales. ' 

Discriminant validity was obtained by' constructiiwjEin a^lterftative, 

negative 'questionnaire for ^ch of the three \bSi^t groups (students, 

teachers, apd administrators). The aife^tr^ati^s^^ 

were identical to the original oneS, expept €hat^^£ch e(|Ciestio^ was 

stated negatively. Subsequently, th^TiAnber^ of items per Iquesfeion- 

n'aire.was to be reduced furthe^l: by field testing ^ach item for^ V>1 

sensitivity ^nd discriminant validity "^during the prit)t study. v ' 



Procedure * ' 

Students: Hal5 of the stvrdents (82) were asked to respojid 
to the original Student Impact Questionnaire, which contained 
sixteen scales (Table 1) • The remaining half 6f tjj^- stiri^ents 
. (l2) were asked to respond -to the alternative Student Impact 



Questionniare^ which contained sixt^een 'identical but negatively. 

^l?9^^t^Ve 

questionnaires are list< 



statedv. scales. Examples of items from origi]^al and^^ 



(Original) 



Lsted^el<^f 






I gaip^ed hew^knq^ledge froiti ^this pr64ij^ 




tu 




^ikjno: new knowled^r^rtom thij^ ' pro 
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insert Tah$,e.l about here 



All ^u^en^s ^^"i^e ,adiaj.n^9tered questionnaires by classroom teachers. 

admini^stiaared p6r each geographical location arid collected on-site. 
Teachers: The same procedure • (stated abov^)*was ufeed; half 



lal amount ^ original "f^j^ft alternative questionnaires ' was 



of the teachers (60X were ask^d to respond to the original Teacher 
Imjfact Questionnai:|^ (Table 2), and the remaining half of the 
teachers" (60) were requested to respond 'to the alternative Teacher 



Impact Questfionnaire. ' The alternative Tea6he^ Impact .Questionnaire 
contained 3^items identical to its original counterparty except that 



6 
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-the items were stated negatively. The procedure far teacher 
data collection differed from student data cpllection in that 

■ • - . 

insert Table 2 about here 



both original and alternative Teacher Impact Questionnaires were 
sent and returned by mail. Sixty-two percent of these teacher 
questionnaires were returned. 

Administrators:- The procedure was repeated, and half of the 
administrators , (28) received the original Administrator Impact 
Questionnaire (Table 3) , while the rest of the administratprs 
(28) responded to the alternative Administrator Impact Question- 
naire. All questionnaires for administrators afs well as teachers 

5_ , 

insert Table 3 about here 

were sent emd returned by mail. Ninety-three percent of the 
Administrator Impact Questionnaires were cbmpleted and returned. 

^ Results 

The basic objective for the previously stated methodology • 
w^s to obtain a low or negative correlation coefficient between 
original and alternative questionnaires for each target group. 



For. students^ the brigipal and alternative questionnaires'^yTelded 
a -.049 discriminant validity correlation^ and an. 83 coefficient d 
of reliability (test-retest) . For teachers, a -.18 discriminant 
valldi^ correlation between original and alternative quest^ionnaires 
was obtained and a ,78 reliability ( test-retest) correlation was-/ s 
found for the original version of the questionnaire. Original and 
alternative questionnaires within the administrator group yielded 
. a -.45 discriminant validity correlation with a .86 ( test^retest) 
/ reliability coefficient for the origina,l questionnairie (see 
Table 4) . 



insert Table 4 about here 



Further adjustments were made with each set of questionnaires 
to increase coefficients of discriminant validity between original 
and alternative forms. A chi-square an^alysis and a Pearson Product- 
Moment -correlation were performed to compare each orginal item in 

the student, teacher, and administrator questionnaires with its 

ft 

. alternative counterpart. The main intention behind these computa- 
tions was to retain only certain items which yielded a signifi- 
cant chi-square value when comparing impact vs. nonimpact ahd * 
which also demonstrated a negative correlation between original and 
alternative statements. 

A description of the chi-square comparisons is shown in 

■t 

Table 5. Table 5 also points out the necessity to collapse the 
. scale points of the questonnaires from five to three due to the 
occurrences of small frequencies. ^ 

8 
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insert Table 5 aUt^out here 




Ideally^ each item sKould also exhibit low or negative correlation 
between the original item arid its alternative facsimile (Tables 
6, 1 , and 8) If both of these requirements were not met by any 
item, the iteta^^wa§_^scarded. The chi-square analysis was performed' 
simply to determine if respondents reacted positively to each 
original' item and negatively to each alternative item. If this 
did not occur/ either the item was difficult to understand or the 
respondents were ambivalent about the question. 

It is evident from Tables , 6,-7, and 8 that many respondents 
answered quite positively on ^ach*original item and very negatively 
on each alternative 'item. However, when alternative and original 
scores were correlated for each item, only ten in each questionnaire 
yielded both a low or negative correlation coefficient, and a 
significant chi-square value \ shown in Teibles 6, 7, and 8) • The ^ 
reason for determining the correlation between original and alternar 
tive scores on each item was to cheeky for consistency of responses. 

, . ^ 

insert Table 6 about here 



insert Table 7 about here 
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Insert Table 8, about here 



Adjusted coefficients of test-retest reliability and discri- 
mi^nt validity for each questionnaire are shown in Table 9. 
The adjustments refer to a reductl?^ of the larger niomber of 
items ^n each questionnaire to the ten which, resulted in a signi- 
ficant chi-square and a low or negative correlation; the reductions 
repres^t final questionnaire versions. ^ / 



insert Table 9 about here 



^, Discussion 

The rather low degree of inter-rater reliability on the 
student and teacher original questionnaires suggests that when 
a pemel of experts is' s,elected, their experience^ should directly 
coincide with the nature of the questionnaire they are ultimately 
going to construct. This is indicated by a fairly high degree 
of inter-rater reliability on the administrator questionnaire. 
The panel of experts were all high level vocational education 
administrators, as. previously mentioned. Time and budget 
constraints did not permit additional representatives of students 
and teachers to be formed into panels of experts. 

The superiority of the administrator impact questionnaire 
over its teacher and student counterparts is evident not only 
in inter-rater reliability # but also in test-retest reliability 

O ' ■ • 10 
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and discriminant validity. In other, words ^ it seems as though 
there is a better assessment of impact in the administrator 
domain than in student and teacher domains, at- least within the 
present study. The Teacher Impact Questionnaire turned opt to 
be ranked second of the three^in terms of validity, b\it very . 
low in inter-rater reliability, indicating that the panel of* 
experts did not really have a consistent conceptualization' of 
impact; as a matter of fact, the teacher questionnaire produced 
the lowest amount of agreement qn items concerning impact. . The . 
Student Impact Questionnaire yielded the lowest coefficient of 
discriminant validity, but did correlate negat-iv^J^ with its 
alternative counterpart, as it was intended to do. 

The most^ sensitive and valid itcipis for student? , turned out 
to be the following (See Table 6) : 

1. gaiping new knowle^ige from the product 

2. understanding the product as it li^ presented 

3. interesting languagfi usag^ in the product 

4. understanding the intent of the product 

5. interest evoked by the product' 

6. available assistance in understanding the product 

7. the value of the material presented in the product 

8. the reality of the *aims of the product 

9. the suitability of the produtt to the individual 
student 

10. ^the opportunity to exchange ideas abd^ut the. product 
Teachers indicated the most sensitive and valid responses concerning 
the following (see Table 7) ; 
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!• the completeness of oe^tails covered in the product 

2. satisfactory guidelines provided ^for use of the product 

3. staff requirements decreased as a result of- the product 

4. feasibility of the product 

, •" ■ • * 

^5., clearly stated objectives v 

6. adequacy of space facilities for using the product 

7. cunple reference and library resources to support 
the product 

8. students asking to use the ptoduct - ' 

• f . 

3. conformity of the product to routines comirianly found 
in a real work situation 

10. reduction in time required to teach a skill 

Among the scunple of administrators , responses on the following 

items were highest in sensitivity and validity (shown in Table 8) : 

J., the absence of many better alternative products 
in the subject area 

^. cost effectiveness of the product 

3. no extra staff needed for implementation 

4. adequacy of available data regarding the proper 
use of the product 

* ^. ease of dissemination of the product 

6. reduction in staff requirements as' a result of the 
product > 

7. freedom to modify the product 

8. high teacher motivation toward the use of the product 

9. endorsement of the product by the local vocational 
advisory committee 

10. timeliness of the product 
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^ ' Generally, the outcomes concerning each questionnaire 
were favorable; it seems reasonably safe to assume that all 
three of them are acceptable in terms of validity. The adjusted 
questionnaires will be the best measures that the researchers 
can take into the field. .Although the^ number of, items (id) 
in eacb of \th^ adjusted questionnaires appears to be small, 
it is a rather accessible number in that the questionnaires 
can be completed in a very small amount of time. These measures 
will contain the most sensitive and meaningful items according 
to the selected ' sample of students, teachers, and administrators 
from the fi^ld of vocational education. 
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PRODUCT IMPACT Ql!)ESTIQNNAIRE FOR STUDENTS 



Name of Product^ 
Directions: 



Everyone has his or her own opinion about this product. 
Some opinions are good, some are bad, and/ others are not 
concerned. There are always different reasons Am: whatever 
opinions ain individual may liave about this products Express 
your opinion of the ||toduct by circling one symbol on th6 
scale for, each* of the following items. Ti^ symbols for" 
the scale are explained below: 



SD 




Strongly disagree 


D 




Disagree 


DK 




Don't know 


A 




Agree 


SA 




Strongly agree 



EXAMPLE: 

i 

I should get an immediate raise "in pay. 

SD D DK A (^Mfk 

All answers are confidential . DO iSBt SIGN YOUR NAME TO THIS QUESTION- 
HA {RE : 

**.«***************************** 

Mr 

y 

1. I gained new knowledge from this product. 
SD D DK A SA 

2. I iinderstand the product as it is presented. 
SD D DK A SA 

3. The language in this product is "alive"; it keeps my interes 
SD D DK A SA 

4. I understand what this product is designed to do. 
SD D DK A SA 

5. This product is interesting. 
SD D DK A SA 

6. This product will help mo reach my vocational goals. 
SD D DK A SA 

EBsLC Original Form ^^16 



s 7.' 



8, 



^Table 1 (^ontinued) 

■ ■ *, • • : ■ if . 

Jhirprodic?!"*^^''''^ " available to help ^ understand 
SD . D DK A SA 

prodult!'""**^"'*^ ^^^""^ presented in this 

SD- D DK Ai 15A , ' . • 

The material presented in this product seems valuable to me. 
SD D . DK A ^ SA ' 

10. The aims of this product are realistic.^' 
SD D DK A . SA 

11. I reaUjst^el that this, product was made for people like me. 
SD y D DK A SA ^ - 



9. 



12. 
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ve an opportunity to exchange ideas with others 
Iroduct. 



SD D DK A SA 

13. This product will help me get a job of my choice. 
SD D DK A SA 

14. This prodact contains ideas /which I can use on the job. 
SD D DK A SA 

15. I enjoy using this product^'. 

/ 

/ 

SD D' DK a SA / 



/ ■ \ 



16. This product has caused ie to think of new ways to do mv 
30b. / , 



SD D DK A SA 



16 
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• ' • ' ' . Table 2 . , • ■ . " ' 

• ■ - • PRODUCT. IMPACT QUESTIONNAIRE FOR 'tEACHERS ' J 

ame^ of Product 4 _ 



. • \ - - ~ s-^ : ■ ^ 

: — ^ V, ■ ^ '■; ■ - ■„.' , 

^. has his or^er -own opinion about "^this product. "^Sorne 

pxnxtfis are good, .some |ire.badv and bthers are not concerned. l«here 
re alvays different reasons for whatever opinion an individual may • 
ave about, this product. Express your opinion of -the product by'' 
irc^ng one symbol? on the scale for ^acK of the following items. ' The 
ymbbls for the scale are ejcplained below: ^ 



SD- 




Strongly disagree 


D 


B 


Disagree 


DK 


m 


Don't know 


A 


s 


Agree 


'SA 




Strongly agree ' 



JCAMPLE! 



r 



0 



I should get an immediate raise in pay. 
SD D DK A (sA) 



answers are confi dential . DO NOT ^IGN YOUR NAME TO THIS QUESTION - 



* * * * * * ** * * * * * 



****** 



The information in this product contains many examples which 
help explain the si2bject matter. 

SD D SK A SA V 'V 

/ . ■ - 

All of the details of the subject matter are covered in this 
product. ' il 

SD D DK A SA 

This product is written in language similar to the language 
used in the world of work. 



SD 



SK A SA 



4. "fheNevents which occur in this product are events which would 
occur in the real world. 

SD k DK A SA 

• - \ y 



O ' 17 

E^iginal Form 



■ . ' J . ■ , ■ ^ . 

' . Table 2 (Continued) 

■ ^ - / ■■ V . . ; ' , ' ■ ' ■ , ^ ' 

The deivelop^s of this \product provided special teachfer \ y 
training* - ) \ f ' • ' " 

SD 5' DK^ A SA ; ' : 

This product mis satisfactory prpcfedures alicSs guidelines for* vse. 

SD • D DK Av SA) . - " a' 

• ■ z^' ^'^ ^- • rv ^^^^ 

Objectives were <5iearly and stif Mciently^ pdinted ertit for; »v 
studen&^Hising this produ^^.* \^ \ ' 



, T^i^prodiict possesses^gopd overall quality (sounds adciirate.^ 
' sof6aally fair, up-tO'-d^te) . • / 



SD ^ fc^ r DK A SA 

Staff -^r-equiremerits decreased as a result of this product. - 
- SD D ' , DK . A SA 

What percentage of this product's original objectives, are 
^being^aCcomplised for the intended group? \ '-^ 

\ 1£ 25 50. 60 75 . 100 " 

< The users of this procKict were allowed to modify it J/ \ 
SD D DK A SA j ' 

This producer harbors creativity and forwand thinking; ^ 

SD D D^: ^ A SA 

This pro<iuct is relevant to the needs of the students 
involved. 



SD, D DK A SA 

This is a feasible product. ¥ 

■ ■ ■ - i ■ 

SD D DK A SA > ' 

There are suggestions ^f or parent-community involvement ftated 
in this product. 

/ . 

SD D DK A SA . 

/ 

This product has clearly stated objectives. ♦ 
SD D DK A SA * 
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■* Stable 2 (6ontinuedl> 



Ifti v^This product con-ttibutfes to the improvement-ef^ci^s^room' 
cliftiate;.or oper^ion. y / . 



.19. This product effective. 
Sp D . PK A /sA 



.A 



20. '•; There is sufficiei/t data available' regaxding 
. - . of this product. . . -«r: 



SD 



DK 



SA 



21. This product contributes to the ijfit^ 
• inot;or-''skills. ' - ^ 



ff perceptual- 



SD 



DK 



22. 



product . 
SD D DK 



A , , SA 

3gpi^WP^^/;g^-^|be?nts who use this 



23, . There'^is no adverse" student -lip^l^^ to this product. 



SD 



24. 



There are adequate space faicl4ities for Effectively usinq 
this product. \ ^ ^ ^ 



$D 



DK / X\ «A 



25. 



There are ample Reference and library resources to support this 
produc.t. . V . * - '-f ■ 



SID 



DK 



SA 



"26. 



27. 



There is f^vp^able student, reaction to this product. 

SD D DKl A "^A ' : ■;■ ; 



students- ask to us^ this pr6<|uct . ■ '^^ r « 
•SD D / DK ,1C\ SA : 



^^^*,k "^^^^ pr^dupt iconfc3(rins :t<i the ":^Ai"i^ne*\commonly found in a real 
=^ work — ' — ■ < ' 



SD 



•r>, DK 



on. 



SA 



■ r 



ERIC 



^5 . 
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, ' i * ^fable 2 (Continued) 



.The 'pTOdutft teaches students to become r4re flexible, 
sb JD-. . ^ A , SA . ' 

The . product prepares students for job level entry. 

The product reia.tes to dif-ferent levels of student abilities. 
SD D' DK A • ' SA 

product provides^ iitunediate feedback to students concerning 
;llieir success or failure. ; 

SQ, ; d dk a sa - » I 

The product helps student?s work together on a common tasK. 

SD ^ D DK A ^ SA _ • 

Students say they like the product. 



SD 



DK SA 



The product reduces the amount of time it takes to teach 
this skill. ^ 

SD D pK A SA ' 

This product is better than similar products I have used in 
•the past. #1 



sfiij^ 



SD D DK A Si 

How lon^'has it been sjflK you first! started to -use this product? 




less th6n 6 mc^nths >^ 1 year ^2 ^ears / 3 years 

longer than 3 ^ea^a X ' 

TotaL number of V^ars^^f t^achin^ experience: 

less than 1 year^ ^ 1-5 years ^ 6-10 years^ more than 

"^10 years. 

Do you have confidence in the person (s) who developed '^his 
product?* 

. y es no don ' t know ' 

My environment is: (please check/^e) j 

Urban (central oity of 50, inhabitants oj\ more) 

Suburban (bordering a central city) 

Rural (at least 25 miles from a central city or a 

population of 5,000 inhabitants or less) . 

f 



Table 3 



y ^ . PRODUCT IM PACT QUESTIONNAIRE FOR ADMINISTRATORS 

Name of Product 

Directions ; 

.Everyone has his or her own opinion about this product 
Some opinions are good, some are bad, and others are not 
concerned. There are always different reasons for^ whatever 
opinion an individual may have about this product. Express 
your opinion of the product by circling one sysbol on the 
scale for each. the following items. The symbols for the 
are explained beTbw: 

SD a Strpngly disagree 

D » Disagree 

DK » Don't know 

A = Agree 

Sa » Strongly a^reie > 



EXAMPLE-; 



I should get an immediate raise in pay. 
SD D . DK A (sa) 
SA^Rl?^''^" confidential . DO NOT SIGN YQUR NAME TO THIS QUESTION - 



********** 



This product contributes to the improvement of school 
organizational efficiency. scnooi 

SD D ' DK A SA 

excessive need for special facilitioa and 
equipment to implement this product. j/§ 

SD D DK A SA 

This product can be transported to other locations. 
SD .D DK A SA ^ 

There are not^ipiy better alternative products available 
m tnis area. 

SD D DK A SA 
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Original Form 



ERIC 



^ 22 



8. 



10. 



Table 3 (Continued) * 

5. Thia product is cost effective. 
SD D DK A SA 

9$"his'p?^2c"' available regarding the dissemi^J.tion 

SD D DK A SA 

7. The implementation of this product does not require extra staff 
SD D DK A SA 

Increased administrative support is not needed for successful 
adoption of this product. • 

SD D DK ^ A , SA 

■ SD D^ DK A SA' 
This product will be used as much or more next year. 
SD D DK A SA 

11. Dissemination of this product is not difficult in our work 
environment. 

SD D DK A SA 
12ii 6taf f «; 



# 6taff «ifeq[uireipents are decreasing as a result of this 
product*^ use. 

SO- D^':f bK A SA 

■1 J," 

13. This prodj^t is being used in large numbers. 
SD D A SA 

14. There is adequate data available regarding this product's 
impact on its^ tflirget audience. f & 

■ 

SD D •< DK V ( A SA 

beJ^g^'SI^Spfel,'"" "^"""^^'^ Objectives are 

1 19.x i£ i£ 25 1 100 
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T^able 3 (Continued) 

f . 

The users of this product are allowed to modify^ it. 

SD D DK A SA 

Teachers say they like the product. 

SD dT dk a SA 

'Te^achers are excited about usin^ the^prod4^ct. ^ 
SD D DK A SA 

The product is^ receiving the endorsement of the local 
vocational advisory committee. 

SD D DK A SA 

The product is timely. 

♦ 

SD D DK A SA 

How long has it been since you first started to use this 
product? \^ 

less than 6 months <v 1 year^ 2 years 

3 years longer than 3 years 

Total nutaber of years of experience: 

less than 1 year; 1-5 years; 6-10 years; 

ro ore than 10 years . 

Do you have confidence in the person (s) who developed this 
product? ^ 

yes no don't know 



My environment is: (please check one) 

Urban (central city of 50,000 inhabitants or more) 

Suburban (bordering a Central city) 

Rural (at least 25 miles from a central city or a popu- 
lation of 5,000 inhabitants or less). 
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Table 4* 

Test-Retest Reliability and Discriminant 
Validity Coeff icirents 
for Impact Questionnaires Before Adjustments 



Type of 
Questionnaiire 



N of Iten^ 



Student 
Teacher 
Administrator 



16 
36 
20 



Test-retest 
Reliability 



.83' 
.78 
.86 



Discriminant 
Validity 



-.049 (n=164) 

- .18 (n»120) 

- .45 (n«56) 
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Table 5 



Example of Collapsing Questionnaire 
Scale Points from Five to Three fot 
^ ^ Chi-Square Analysis , 



Basic five scale points: 





SD 


D 


DK 


A 


SA 


Original 


2 


9 


9 


8 


0 


Alternative 


0 


7 


13 
— A 


7 


1 



Scale points collapsed from five to three due to small lllrequencies 





D 


PK 


A 


Original 


11 


9 


8 


^ Alternative 


7 


13 , 


8 
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Table 6 ^ 

f 

Correlation and Chl-Square Values between the 
Original and Alternative Student Impact Questionnaires" 



item 



Correlation (df«80) Chi-Sguare (df»2} 



1. " I gained new knpwledge 

from this product. 

2. I understand the product 
as It .Is presented. 

3. The leuiguage In this 
-product Is "alive"; It 

keeps my interest. 

4. I understand what this 
product is designed to 
do. 

5. This product is inter- 
esting . 

6. This product will help 
me reach my vocational 
goals. 

7. If needed, assistsmce 
is availeJDle to help 
me under ^'^ahd this 
product . 

8. I am stimulated to 
think about the topics 
presented in this 
product. 

9. The material presented 
in this product seems 
valuable to me. 



-.01 

-.19 

-.11 

-.10 
-.01 

.23 



-.05 



.02 



-.02 



106.01 

54.82 

39.24 

123.56 
117.70 

110.97 



105.29 



58.60 



101.54 
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Table 6 (Continued) 



10 . The aims of this product 
are realistic. 

11. I really feel that this 
^ product was made for 

people like me. 

12. usually have an 
opportunity to exchange 
ideas with others about 
this product. 

13. „ This product will help 

me get a job of my 
choice. 

14. This product contains 
ideas which I can use 
on the job. 

15. I enjoy using this 
product . 

16. This product has caused 
me to think of new^ ways 
to do my job. 



-.38 



"^.02 



-.02 



.06 



.02 

r 

.03 



.10 



114.50 



76.38 



45.40 

59.87 

115.56 
69.00 

64.66 



All reported chi-9quare values are statistically significant 
beyond the .001 level. 

+This item was retained for the final version of the. Student 
Impact Questionnaire. 
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Table 7 \ 

Correlatiojtji^ and Chi Square Values between the Original 
and Alternat;|.ve Teacher Impact Questionnaires 



Item <lorrelation(df=58j thi-Square (df»2) 



1. The information in 
this product contains 
many examples which 
help explain the sub- 
ject matter. 

2. All of the details 

of the subject matter 
are covered in this 
product. , 

3. This product lis written 
in langiiage simij^ar to 
the language used in 
the world of work. 

4. The events which occur 
in this product are v 
events which would 
occur in the real world.. 

5. The developers of this 
product provided special 
teacher training. 

6. This product has satis- 
factory procedxires cmd 
guidelines for use. 

7. Objectives were clearly 
and sufficiently pointed 
out for students using 
this product. 

8^ This product possesses 
good overall quality 
(sound, accurate, soci- 
ally fair, up-to-date) . 



.17 . 89.96*** \ 

\ 

-.02 44.90*** 

.26 110.36*** 

.14 111.26*** 

.23 68.68*** 

-.05 112.12*** 

.08 111.26*** 

.07 109.28*** 



?9 



I 

Table 7 (Conti^^^^j 



+ 9 . Staff requirements 
decreased as a 
result of this 

product. -.i3 7.73^ 

10. What percentage of 
this product's ori- 
ginal objectives are 
being accomplished 
for the intended 

group? .15 67.61*** 

11. The users of this \ 
JWRJduct were allowed 

W modify it. .09 98.36*** 

12. This product harbors 
creativity and for- 
ward thinking. -Oq 83.40*** 

13. This product is 
relevant to the needs 

of the students involved. 113. 39*** 

+ 14. This is a feasible 

product. .120.00*** 

15. There eu^e suggestions 
for pareht- community 
involvenient states in 

this product. .2o , 70.15*** 

+ 16. This product has clea^^' 

ly stated objectives. ^.ig 120. 00*** , 

17. This product contributes 
to the improvement of 

i§ learning. .0$ 120. 00*** 

18. This product contributes 
to the improvement of 
classroom climate or 

operation. .O9 105. 10*** 
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' ^ Table 7 (Continued) 

19. This product is 

effective. .08 ^ 109.60*** 

20. There is sufficient 
data available regard- 
ing student impAtst of 

this product. .02 20.96 *** 

21. This product contri- 
butes to the iinprpve- 
inent of perceptual 

motor skills. .10 67.32*** 

22. There aure measured 
cognitive gains for 
students who use 

^his product. .08 91.09*** 

23. There ^s adverse t • 
student reaction to 

this product .05 14.94*** 

+ 24. There eure adequate / 
space facilities f6r 
effectively using 

this product. -.13 85.40*** 

+ 25. There are aiqple 

reference and library 
resources to siqpport 

this product. -.02 47.59*** 

26. There is favorably 
student reaction to. 

this product. .09 82.90*** 

+ 27. Students asic to use 

this product. -.01 15.17*** 

+ 28. The product conforms 
to the routines com- 
^ monly found in a real 

. work situatior^. -.16 106.80*** 
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Table 7 (Continued) 



29. Th« product teaches 
students to become 

iDore flexilile. .12 

30. The product prepares 
students for ^Ob 

level entry. .03 

i 

31 e The product relates 

to different levels ^ 

of student abilities • .26 

32 e The product provides 
iiomediate feedback to 
^tudents concerning 
their^success or 
failxire. .15 

33. The product helps 
students work together 

on a consnon task. .16 

4. 

34. Students say they like 

the product. .11 

35. The product reduces 
the amount of time it 
takes to teach this > 
skill. -.11 

36. This product is better 
than similar products 

I have used in the past. .12 



76.72*** 



63.80*** 



54.10*** 



78.80*** 



87.00*** 
52.66*** 



24.94*** 



61.16*** 



*** Significant beyond the ,001 level 
* Significant beyond the .05 level 

+ This item was retained for the final version of the Teacher 
Iidpact Questionnaire 
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Table 8 

Correlation and Chi Square Values betv;een the Origfinal 
^ «tnd Alternative Administrator Questionnaires 

— f ■ ■ — — 

Item Correlation (df«2 6) Chi-Square(df=2) 



!• This product contributes 
to the improvement of 
school organizational 

efficiency. .28 24.58*** 

^2• There is no excessive 
need for special 
facilities and equip- 
ment to implement this 

product. .23 44.70*** 

3. This product can be 
transported to other 

locations. .12 46.68*** 

4. There are not many 
better alternative 

' products available in 

thi« area. -.08 . 30.80*** 

' J- ■ 

5«. Thi'S product is cost 

; efftective. .02 34.84*** 

6. ' Inhere is adequate 

s dfita, av/iilabe regard- 
ing the dissemination 

of this product. < .08 22.64*** 

7. The implementation 
of this product does 
not require extra 

stiff. -.01 20.98*** 

8. Inc;teasei^ admini- 
strative support is 
ndt heeded for success- 
tul ijdoption of this 

, product. \ . .15 .50 
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Tables (Cpntinued) 

4- $. Ttimxm la ad«quat« data 
available regarding r 
. t>a propar viae of I • , 70*** 

thia product. -.02 3lr 

., ' ■ ; ...>:; ■ ♦ . . 

10. Tliia product will 

be uaed ae mwcfe ' ,3*** 

.or more next year. .21 43-'-' 

* 11. Diaaeinihation of 
thia product ia 
aot difficult in 

our woj^ environ- nn**i 
ment. ^ -.19 43-20 

■«- 12. Staff requirements 

are decreasing I 

as a result of f og*** 

^ this product's use. -.37 20'^° 

13. This product is 

being used in large ^4*** 
numbers. .05 25*^ 

14. There ia adequate 
data available 
regarding thia 
product* a inpiiet 

on ita target ,n 
audience. -.13 1.^" 

15. Mbat percentage 

of thia product'a ' 
original objectivea ^ 
are being accom- " 9 



pliahed? .07 38.8*' 

66*** 



^ 16. The uaera of this 

product are allowed 

to modify it. -.03 29 



17. Teachers say they a a*** 

like the product. .16 33-^^ 



Table 8 (Continued) . 

» . .. ' . . ■ ' 

^18. t4,ich«ra ar« excited 

'«bbut using the 
^. Product. -.05 21.32*** 

^ V"*- The product is re- 

] 9*iving the ^ 
•thdoraeimsnt of the C 
, /; local vocational 

*dvlsory conrmlttee. -.15 " 43.28*** 

+ The product is 

tiinely. . -.37* 52.14*** 



**:* Significant beyond the .001 level 

+ This item was retained for the final version of the Administrator 
Impact Questionnaire . • 




Table 9 



Test-yetest Reliability and Discriminant 
validity Coefficients 
for Adjusted Impact Questionnaires 



Type of s Test-'retest Discriminant 

Questionnaire N of items Reliability Validity 



Student 10 .79 -.11 (n=164) 

Teacher 10 .71 -.24 (n'=120) 

Administrator 10 .80 -.57 (n=56) 
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